Encoder Optimization of Stereoscopic Video Delivery Systems

ABSTRACT

Controlling a feature of an encoding process for regions of an image pattern representing more than one image when the regions include an amount of disparity in the represented images that would result in cross-contamination between the represented images if encoded with the feature. The control may be, for example, any of turning the encoding feature off, using the encoding feature less often than when encoding an image pattern representing a single image, negatively biasing the encoding feature, and enabling the encoding feature for regions determined to have zero or near zero disparity and disabling the feature for all other regions. The represented images comprise, for example, any of a stereoscopic view, multiple stereoscopic views, multiple views of a same scene, and multiple unrelated views.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationNo. 61/082,220, filed 20 Jul. 2008, hereby incorporated by reference inits entirety.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND OF THE INVENTION

1. Field of Invention

The present invention relates to Video coding and more specifically tomulti-image and stereoscopic video coding.

2. Description of Related Art

In recent years, content providers have become interested in thedelivery of stereoscopic (3D) content into the home. This interest isdriven by the increased popularity and production of 3D material, butalso the emergence of several stereoscopic devices that are alreadyavailable to the consumer. Although several systems have been proposedon the delivery of stereoscopic material to the home that combinespecific video view “arrangement” formats with, primarily, existingvideo compression technologies such as ISO MPEG-2, MPEG-4 AVC/ITU-TH.264, and VC-1, these systems do not provide any information on how thevideo encoding process should be performed. This has consequentlyresulted in poorly designed stereo video encoding solutions with subparperformance, which has been detrimental in the adoption of such systems.

SUMMARY OF THE INVENTION

The present inventors have realized the need to develop efficient codingsystems for multi-image video (e.g., multi-view and stereoscopic or 3Dvideo) and other new features and be compliant with standard videocodecs (e.g., monoscopic video compatible codecs). In one embodiment,the present invention provides a method of encoding images comprisingthe step of controlling an encoding feature of an encoding process forregions of an image pattern representing more than one image to beencoded by the encoding process when the regions include an amount ofdisparity in the represented images that would result incross-contamination between the represented images if encoded with thefeature. The step of controlling an encoding feature comprises, forexample, at least one of turning the encoding feature off, using theencoding feature less often than when encoding an image patternrepresenting a single image, negatively biasing the encoding feature,instituting a relative degree of change in the encoding feature wherethe relative degree of change is based on the disparity, at leastpartially enabling the encoding feature for regions determined to havezero or near zero disparity and disabling the feature for all otherregions, and enabling the encoding feature for a region havingrelatively low or no stereoscopic disparity when compared to acorresponding region of another image in the pattern, disabling theencoding feature for a region having relatively higher stereoscopicdisparity, enabling the encoding feature for regions in backgroundlocations and disabling the encoding feature for regions in foregroundlocations, and at least partially enabling the feature for regionsdetermined to have zero or near zero motion and disabling the featurefor all other regions.

The represented images may comprise, for example, at least one of astereoscopic view, multiple stereoscopic views, multiple views of a samescene, and multiple unrelated views.

In one embodiment, the method further comprises the step of classifyingthe disparity of a region as being one of high disparity, low disparity,and zero disparity, and using the disparity classification to direct thestep of controlling. In one embodiment, the amount of disparitycomprises an amount of disparity between neighboring regions in therepresented images. In one embodiment, the amount of disparity comprisesan amount of disparity between corresponding regions in the representedimages. In one embodiment, the amount of disparity comprises an amountof stereoscopic disparity. In one embodiment, the disparity isdetermined by bandpass filtering the images, and the bandpass filteringmay be applied both vertically and horizontally. In one embodiment, thedisparity comprises differences in a location of the regions. In oneembodiment, the amount of disparity comprises an amount of illuminationchange in the corresponding regions, where the corresponding regionscomprise, for example, corresponding regions in a stereoscopic pair ofimages. In one embodiment, the amount of disparity comprises an amountof motion estimation between the images.

In one embodiment, the amount of disparity comprises an amount ofestimated motion in corresponding regions of the more than one image.The motion estimation comprises, for example, at least one of a featurebased motion estimation (e.g., an Enhanced Predictive Zonal Search(EPZS)), a pixel based motion estimation, a block based motionestimation, phase based motion estimation, frequency domain based motionestimation, pixel recursive motion estimation, true motion region motionestimation, and Bayesian based motion estimation.

In various embodiments, the encoding feature comprises at least one ofde-blocking, transform, and quantization, motion compensation,inter-frame prediction, intra-frame prediction, a color format andsampling configuration. The encoding process comprises, for example, avideo encoding process. In one embodiment, the encoding processcomprises High Definition (HD) video encoding. In one embodiment, theencoding process comprises scalable video coding. The scalable videocoding comprises, for example, the scalable video coding extension ofthe AVC/H.264 standard. In various embodiments, the encoding processcomprises any video codecs such any of MPEG codecs, includingMPEG-1/MPEG-2, MPEG-4 Part 2, MPEG-4 AVC/H.264, other proprietarilycodecs AVS, VC1, RealVideo, On2 VP6/VP7, or other coding techniquesincluding the Multi-View Coding Video extension of AVC, Motion JPEG andMotion JPEG-2000.

The image pattern comprises, for example, a “black” and “white”checkerboard arrangement of pixels where the “black” pixels comprisepixels of an image of a first channel in a stereoscopic view and the“white” pixels comprise pixels of a second channel in the stereoscopicview. In essence, for example, the image pattern comprises a“multi-colored” “checkerboard” arrangement of pixels where eachindividual “color” of the “checkerboard” comprises pixels of anindividual one of the more than one images. The “multi-colored”“checkerboard” may comprise, for example, more than two “colors.” Statedmore generally, the image pattern comprises a “multi-colored”arrangement of pixels where each individual “color” of the arrangementcomprises pixels of an individual one of the more than one images. Thearrangement may comprise, for example, any of at least one of rows andcolumns allocated to each “color”/image, and a modulo-spacingarrangement of locations in the arrangement allocated to each“color”/image.

In various embodiment, for example, the encoding process comprises oneor more of field based encoding, and field based encoding performed inat least one of a picture level and a macroblock level.

The amount of disparity may be computed, for example, via a processcomprising at least one of a picture level analysis, analysis of aslice, a region level analysis, a macro-block, and block level analysis.In one alternative, the amount of disparity may be determined in a stepcomprising computing distortion. Computing distortion may comprise, forexample, at least one of separating stereo view samples and computingdistortion based on a 3D view of the separated samples, comparing theoriginal images to images represented in the “checkerboard” after beingdecoded, comparing original pre-image pattern images to the imagesrepresented in the image pattern after being decoded. The computeddistortion comprises, for example, at least one of luma and chroma, acomparison to at least one of the images prior to being represented inthe image pattern. In yet another alternative, disparity itself isdetermined in a step comprising computing distortion wherein thedistortion is computed from an original pair of 3D images that aresubsequently encoded into the image pattern and then comparing theoriginal pair of images to the images decoded from the image pattern.

The method may further comprise, for example, a step of reducing qualityof at least one image set of the represented images. Reducing qualitymay be performed, for example, based on a content of the image set beingreduced, and/or based on a pricing model that takes into account adesired and acceptable level of quality of the image sets. In variousembodiments, the image sets within the represented images areprioritized based on a desired quality, and the step of reducingquality, if performed on an image set, is regulated according to theimage set priority such that a lower priority image set is reduced inquality further than a higher priority image set. Further intelligencein reducing quality may also be utilized, for example, where the imagesets within the represented images are prioritized based on a desiredquality, and the step of reducing quality, if performed on an image set,is regulated according to the image set priority and size of the imageset such that a lower priority image set is reduced in quality furtherthan a higher priority image set unless a higher priority image set istoo large and additional space or bandwidth is needed to carry all ofthe image sets. The reduction in quality need not be uniform across animage set, that is it could be performed only on a region or amacroblock/block of an image set, and may be performed, for example, ona view of a stereoscopic image set.

Further, the reduction in quality may be performed in a subregion of animage and not for the entire image. For example, this could be done onlyfor a region with high texture but not anywhere else. Alternatively, wecould have within the same image pixels corresponding to oneview/sub-image in a region being of higher quality and vice versa forother regions

The invention may also be embodied in a method comprising the steps of,receiving a stereoscopic pair of images, combining the images into animage pattern, evaluating at least one tool for encoding the imagepattern, and applying at least one of the encoding tools in an encodingprocess if an amount of cross-contamination between the images in eitherthe encoding or corresponding decoding processes is below apredetermined cross-contamination threshold, wherein the encodingprocess comprises an existing video format. The step of applying atleast one of the encoding tools comprises, for example, prioritizingregions of the images and applying the encoding tools to each regionaccording to the priority. The prioritization comprises, for example,high priority regions where at least one encoding tool is applied andlow priority regions where the coding tools are not necessarily applied.In one alternative, the prioritization is based on a region passing atleast one predetermined threshold. The at least one predeterminedthreshold comprises, for example, at least on of a stereo disparity,motion detection, luminance difference, and chrominance differencethresholds. The threshold/thresholds may be fixed/predefined by theapplication or characteristics of the encoding process, random (orpseudorandom) again given the application, or adaptive given the contentor earlier decisions performed. Decisions could be driven by complexity,quality, and of course correlation of neighboring (both temporally andspatially) samples. The existing video format comprises, for example, avideo format not developed for encoding stereoscopic images, and may be,for example, a scalable video coding extension of the AVC/H.264standard.

The step of evaluating comprises, for example, at least one of stereodisparity analysis, motion estimation analysis, luminosity analysis,analysis from multi-pass encoding, analysis from pre-processing andearlier passes, and Motion Compensation Temporal Filtering (MCTF)analysis. The at least one tool comprises at least one of a de-blockingtool and a prediction tool.

This method may further comprise the step of applying at least one ofthe encoding tools, and may yet further comprises applying at least oneof the encoding tools at a reduced level if the amount ofcross-contamination is within an acceptable pre-determinedcross-contamination range. Cross-contamination comprises, for example,at least one of luma, chroma, blocking, and stereo view contamination.

The step of evaluating comprises, for example, a cost process. The costprocess may comprise, for example, a function of computational costs andcontamination costs. The cost process may comprise a lagrangian costprocess or costing process.

This method may further comprise the step of exploiting Stereo ViewMasking (SVM) by applying at least one of the encoding tools in anencoding process in cases where μ_(LV)>μ_(RV), or μ_(LV)<μ_(RV)resulting in a higher bias towards the quality of one view. In onealternative, exploiting stereo view masking (SVM) may be embodied byproviding a lower quality version of one of the stereoscopic pair ofimages and alternating the lower quality version between subsequent leftand right views of the stereoscopic pair of images. In yet anotheralternative, exploiting stereo view masking (SVM) may be embodied byproviding a varying quality of at least one of the stereoscopic pair ofimages wherein the quality is selected based on a priority of the image.In still yet another alternative, exploiting Stereo View Masking (SVM)may be embodied by applying at least one of the encoding tools in anencoding process if an amount of cross-contamination in one of theimages in either the encoding or corresponding decoding processes isabove a first predetermined SVM cross-contamination threshold and anamount of cross-contamination in the other image in either the encodingor corresponding decoding processes is below a second predetermined SVMcross-contamination threshold.

The invention may also be embodied as an encoding device, comprising, aninput port configured to receive a bit pattern comprising image data tobe encoded, an encoder comprising a set of tools configured to encodeaspects of the image data to be encoded, and an evaluation processorconfigured to, evaluate at least one factor between correspondingregions of a set of images embedded in the bit pattern of the image tobe decoded, and regulate an amount of use of at least one of the toolsbased on said evaluation. The factor comprises, for example, at leastone of disparity, luminance, chrominance, and motion estimation. The setof tools comprises, for example, at least one of a de-blocking tool anda motion prediction tool. The encoder comprises, for example, a scalablevideo encoder, and may further comprise an extension of an existingvideo format.

In one embodiment, the video format comprises AVC/H.264. In anotherembodiment, the video format comprises one of AVS and VC1.

Regulation by an evaluation processor may comprise, for example,negatively biasing at least one of the tools if at least one of theevaluated factors exceeds a pre-determined threshold. In onealternative, regulation by the evaluation processor comprises negativelybiasing at least one of the tools by an amount that varies depending ona priority level associated with the evaluated factors. In anotheralternative, regulation by the evaluation processor comprises reducingan effect of use of at least one of the tools if at least one of theevaluated factors exceeds a pre-determined threshold.

In one embodiment, the set of images comprises one of a stereoscopicview, multiple stereoscopic views, multiple views of a same scene, andmultiple unrelated views.

Portions of both the device and methods, and other embodiments may beconveniently implemented in programming on a general purpose computer,or networked computers, and the results may be displayed on an outputdevice connected to any of the general purpose, networked computers, ortransmitted to a remote device for output or display. In addition, anycomponents of the present invention represented in a computer program,data sequences, and/or control signals may be embodied as an electronicsignal broadcast (or transmitted) at any frequency in any mediumincluding, but not limited to, wireless broadcasts, and transmissionsover copper wire(s), fiber optic cable(s), and co-ax cable(s), etc.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete appreciation of the invention and many of the attendantadvantages thereof will be readily obtained as the same becomes betterunderstood by reference to the following detailed description whenconsidered in connection with the accompanying drawings, wherein:

FIG. 1 is a diagram illustrating row sequential/field interleavedarrangement for the delivery of Stereoscopic material;

FIG. 2 is a diagram illustrating Stereo view reconstruction of fieldinterleaved pictures;

FIG. 3 is a diagram illustrating a checkerboard interleaved arrangementfor the delivery of stereoscopic material according to an embodiment ofthe present invention;

FIG. 4 is a diagram illustrating Stereo view reconstruction ofcheckerboard interleaved pictures according to an embodiment of thepresent invention;

FIG. 5 is a diagram of a video encoder (i.e. AVC encoder) configuredaccording to embodiments of the present invention;

FIG. 6 is a diagram of a video decoder (i.e. AVC decoder) capable ofdecoding optimized coded images/video according to an embodiment of thepresent invention;

FIG. 7 is a diagram illustrating view disparity analysis andclassification using Motion Estimation according to an embodiment of thepresent invention;

FIG. 8 is a diagram illustrating a view disparity analysis andclassification of CB interleaved pictures using band pass filteringaccording to an embodiment of the present invention;

FIG. 9 is a diagram illustrating a view disparity analysis andclassification using band pass filtering according to an embodiment ofthe present invention;

FIG. 10 is a diagram illustrating an example view disparity analysis andclassification of CB interleaved pictures using Motion Estimationaccording to an embodiment of the present invention;

FIG. 11 is a flow chart illustrating de-blocking parameter determinationaccording to an embodiment of the present invention;

FIG. 12 is a diagram illustrating 4:2:0 content according to anembodiment of the present invention. The diagram illustrates that modulo4 integer samples have no cross view contamination for motion estimationand compensation purposes. Limiting motion estimation to those positionscan have considerable impact on both coding performance and complexity;

FIG. 13 is diagram illustrating a field based encoding according to anembodiment of the present invention; and

FIG. 14 is a block diagram illustrating a coding mode decision accordingto an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Stereoscopic video formats may include row interleaved (fieldsequential), checkerboard, pixel/column interleaved, side by side andothers, which by nature result in very different image characteristicscompared to normal images and for which such generic video codecs havebeen originally optimized. Generic video codecs such as AVC/H.264 assumethat all pixels within an encoded picture belong to the same image (orin this case view) and have certain spatio-temporal characteristics thatcould be exploited using techniques such as spatio-temporal prediction,e.g., motion estimation and intra prediction, transform andquantization, and de-blocking among others.

These characteristics, however, may not always exist within these stereoimage formats impairing or restricting the performance of such tools. Inparticular, the row interleaved approach arranges two stereo view imagesinto a single image by the use of appropriate vertical filtering anddecimation prior to compression as can be seen in FIG. 1.

Similarly the reconstruction process involves extracting thecorresponding pixels from each view from the combined image, andgenerating the missing samples using appropriate filters as can be seenin FIG. 2. This suggests that spatial correlation among verticallyadjacent pixels is considerably lower and special care should be takenwhen encoding such pictures so as to not contaminate the information ofone view with that of another. If an encoding solution does not adhereto such rules, this may result in artifacts that could be rather visiblewhen viewing the content with an appropriate stereoscopic display.

A checkerboard interleaving method (e.g., FIG. 3 & FIG. 4), would resultin similar or even more emphasized problems. However, the checkerboardmethod is of particular interest because of improved resolution comparedto other formats, and since it is currently supported by availabledisplay devices. The present inventors have realized the need for thecreation and adoption of a stereoscopic video format for consumerapplications assuming, however, the creation of appropriate videoencoding systems with acceptable quality performance. The pixelarrangement, however, presents increased challenges to existing videoencoders, which should be considered during the encoding process.

The present invention encompasses several methods, devices, andstructures that may be embodied in a variety of ways, each of which maybe considered during the encoding process which may result in eitherimproved coding performance, measured in increased quality given a fixedbit-rate or equivalently reduced bit-rate given a fixed quality target,or/and in complexity reduction of such encoding solutions with little ifany impact in coding performance. The methods presented may be appliedto any present or future video codec, including scalable solutions suchas the scalable video coding extension (SVC) of AVC/H.264, but alsoother arrangements which may dictate the arrangement within a singlepicture of more than 2 views, i.e., for multi-view video coding.

Methods of the present invention are based primarily on the principlethat all available encoding tools should be considered in a way thatevaluates any possible contamination of the samples of one view fromthose of another. De-blocking in particular, usually performs afiltering process across adjacent samples, which, given certainstereoscopic arrangements such as the field and checkerboard interleavedmethods, could considerably impair the stereo experience. It istherefore proposed that de-blocking is disabled or appropriatelycontrolled for such material or for regions with significant stereo viewdisparity. For example, FIG. 11 is a flow chart illustrating ade-blocking parameter determination. The parameter is determined basedon, for example, a given disparity classification process.

Determination of the extent of stereo view disparity may be performed,for example, by using a basic motion estimation test between the twoviews. The test could be performed using a block based motion estimationscheme, which may also include illumination change characterization,such as the Enhanced Predictive Zonal Search (EPZS) process, or othermethods including pixel-recursive, phase based, and true-motion regionbased motion estimation among others (For example, FIG. 7 is a diagramillustrating view disparity analysis and classification using MotionEstimation according to an embodiment of the present invention). Usingsuch processes, regions determined of having zero or near zero motioncould be categorized as zero stereo view disparity or low stereo viewdisparity regions, while all others are categorized as high stereo viewdisparity regions.

In one embodiment, de-blocking could be permitted and appropriatelyadjusted only on zero or low disparity regions, and disallowed orappropriately adjusted with a lower strength de-blocking mechanism onregions with high disparity. This could be done for example usingappropriately arranged slices within the image and associating each onewith the necessary de-blocking signals. Regions could also becharacterized using other mechanisms such as the use of band passfilters such as the one presented in M. Latzel, J.K. Tsotsos, “A robustmotion detection and estimation filter for video signals,” inProceedings of the 2001 IEEE International Conference on ImageProcessing (ICIP'01), vol. 1, pp. 381-384, Oct. '01, and extended in P.Yin, A. M. Tourapis, J. Boyce, “Fast decision on picture adaptiveframe/field coding for H.264,” VCIP 2005. Instead of, however, applyingsuch filters only vertically, given the stereo view arrangement, thesame filter could be applied both vertically and horizontally and thendetermining if a region could be classified as a high, low, or zerodisparity region. For example, FIG. 8 is a diagram illustrating a viewdisparity analysis and classification of CB interleaved pictures usingband pass filtering according to an embodiment of the present invention.

In one embodiment this is done by extending these methods to support 2Dfilters. These filters could also be applied by sub-sampling the imageeither horizontally and/or vertically, and by applying the appropriatefilters given the orientation of the sub-sampling process. For example,we may create two sub-sampled versions of the original stereomultiplexed image, the first using horizontal sub-sampling, and thesecond using vertical sub-sampling. The first version is then processedusing vertical band pass filtering, while the second one is processedusing horizontal band pass filtering. Samples in both images are thencharacterized given the behavior of these filters and their results areaccumulated to categorize the entire image. For example, FIG. 9 is adiagram illustrating an example view disparity analysis andclassification of CB interleaved pictures using two stage band passfiltering according to an embodiment of the present invention.

Other methods for categorizing disparity such as that described inAdams, “Interlace motion artifact detection using vertical frequencydetection and analysis,” U.S. Pat. No. 6,909,469, the contents of whichare incorporated herein by reference in their entirety, could also beused. It should be noted that motion based analysis could be used evenfor already interleaved picture arrangements by either performing theanalysis through decomposing and reconstructing the final, fullresolution views, or by creating sub-images of lower resolution thatcontain samples from only one view each and performing motion estimationand characterization on these sub-images. For example, FIG. 10 is adiagram illustrating an example view disparity analysis andclassification of CB interleaved pictures using Motion Estimationaccording to an embodiment of the present invention. A correspondingprocess employs a view decomposition stage which may involve, forexample, filtering or the consideration of only aligned samples (i.e.samples only from even or odd lines in the image).

It should be noted that the above disparity analysis is not only usefulfor the selection of the de-blocking mechanism that is to be applied toeach region of an image, but also for other purposes such as thedetermination of which coding tools, such as motion estimation, shouldbe considered for encoding a region, which may considerably help inreducing encoding complexity.

In one embodiment, an encoder first performs disparity analysis of animage. If a region is classified as a high stereo view disparity region,tools that may result in contamination of samples of one view with thoseof another are disabled or reduced in consideration priority during theencoding decision process. For example, it is observed that sub-samplemotion compensation in existing codecs is performed using a variety offilters that are applied across the image. In particular, in AVC/H.264 a6-tap filter is employed for half sample positions, and bilinearfiltering is employed for quarter-sample and chroma sub-samplepositions. Other filters, including bicubic, 2D non-separable, diagonal,and others may also be used.

For high stereo view disparity regions the likelihood of sub-sampleinterpolation is considerably lower, and in fact, if used, may result inconsiderable coding artifacts that the residual coding process may needto correct. Therefore, the encoder may determine that it is unnecessaryto consider such tools for these regions, considerably reducingcomplexity and the possibility of incurring coding artifacts. Similarconclusions could also be made for the consideration of certain integersample positions within the image. In particular, it is observed that4:2:0 content, and the CB interleaving pattern, only modulo 4 integersample positions compared to the current position avoid anycontamination from a different view. For example, see FIG. 12 which is adiagram illustrating 4:2:0 content according to an embodiment of thepresent invention. The diagram illustrates that modulo 4 integer sampleshave no cross view contamination for motion estimation and compensationpurposes. Limiting motion estimation to those positions can haveconsiderable impact on both coding performance and complexity.

In particular positions of distance 1 horizontally or vertically resultin obvious prediction contamination of the luma component, but alsopartial contamination of the chroma component (because of the use ofbilinear filtering for chroma). Similarly for positions of distance 2horizontally or vertically, even though luma sample prediction isproperly performed, are still affected by improper prediction of chromasamples.

In one embodiment, and similar to the sub-sample case, an encoder doesnot consider or reduces the consideration priority (i.e., within a fastmotion estimation scheme such as EPZS) samples that may result in crossview contamination during the encoding process. Consideration prioritycould be designed in a way such that we first consider module 4 integerpositions. If the best position using these samples does not satisfy acertain thresholding criterion, e.g. distortion such as SAD is not belowa certain value T1 then all other positions are also examined. Theremaining positions could also be examined in stages based on stereoview sample contamination impact, i.e. first examine modulo distancepositions (quality impact from chroma contamination only), followed byodd distance positions (contamination from both luma and chromasamples).

In one embodiment, this consideration could be applied only at apre-analysis/first encoding pass stage, i.e., when pre-analysisstatistics are required for the determination of encoding parameters,such as quantization parameters, in a multi-pass encoding scheme. In adifferent embodiment, this approach is only considered for certainreference pictures or block sizes of a certain size (i.e., for AVC/H.264only for block sizes 8×8 and below) and not for larger partitions forwhich all samples are examined. The approach can also be extended forstereo view multiplexing schemes that consider alternating subsamplingfor every picture. It should be obvious to anyone familiar with the artthat this approach could be easily extended to other color and/or chromasubsampling formats. Furthermore, the above discussions can also beextended to intra prediction.

Field based coding, even if the stereo view samples are not arranged ina row interleaved (field sequential) approach, could still be useful forencoding purposes and could provide additional benefits. In particular,for the checkerboard interleaving approach, field coding, both at thepicture and macroblock level, would result in images which full verticalstereo view correspondence which may improve compression efficiency.This would also improve the reliability of vertical interpolation,which, given the previous discussions, can now be more useful and morereliable for motion and intra prediction (i.e. improved performance ofvertical intra prediction). Impact on stereo view quality can also beconsidered during the design and selection of quantization matrices forboth frame and field coding arrangements.

FIG. 13 is diagram illustrating a field based encoding according to anembodiment of the present invention. The field based coding is both atthe picture and macroblock level and is useful for various types ofstereo view mixing such as checkerboard interleaving. In the illustratedexample arrangement only horizontal sample contamination existspotentially improving coding efficiency.

In one embodiment, coding performance can be considerably improved byperforming encoding decisions by considering the final output format,i.e. final stereo view image including any appropriate interpolation,for evaluating the distortion of a given decision. Decisions couldinclude picture and macroblock level decisions such as the selection ofa certain block coding mode, reference, motion vector and/orillumination change parameters, intra prediction mode among others.Distortion, such as the sum of absolute differences (SAD) or sum ofsquare errors (SSE), can be computed compared to either the original noninterleaved stereo views, or the reconstructed stereo views using theoriginal, non-encoded stereo interleaved images. For, each coding mode,distortion is evaluated by considering the prediction (for lowcomplexity or basic motion estimation) or the final reconstructed (forhigh complexity) samples and de-multiplexing and appropriatelyinterpolating the views.

Interpolation could be using basic interpolation processes (i.e.averaging of all adjacent same view samples), or could be done usingmore complex processes that emulate the processing performed priorand/or during display (e.g., see FIG. 14 which is a block diagramillustrating a coding mode decision according to an embodiment of thepresent invention. The coding performance of a mode decision and/ormotion estimation can be improved if distortion is computed byseparating stereo view samples and computing distortion given the 3Dviewing experience) In particular, assuming the usage of lagrangianoptimization, video coding decisions can be performed using thelagrangian cost of the form:

J(λ)=μ_(LV) D _(LV)(R)+μ_(RV) D _(RV)(R)+λ·R  (1)

where DLV and DRV are the distortion values for the left and right viewrespectively, μ_(LV) and μ_(LV) are the lagrangian multipliersassociated with each view, R is an estimate or the actual bit-rate forselecting that coding mode.

Finally, λ is the lagrangian parameter for the rate parameter. In oneembodiment μ_(LV)>μ_(RV), giving higher bias towards the quality of oneview (i.e., left) compared to the other (i.e., right) which can helpexploit stereo view masking characteristics. This method may also bedone while considering the disparity region analysis method describedearlier. In particular, for example, for zero or low stereo disparityregions we can jointly consider distortion of all views without havingto perform any additional processing, and only consider this method forhigh stereo disparity regions therefore reducing complexityconsiderably.

Based on the present disclosure, various devices of varying capabilityand efficiency may be constructed. An exemplary such device isillustrated, for example, in FIG. 5. FIG. 5 is a diagram of a videoencoder (i.e. AVC encoder) configured according to embodiments of thepresent invention, and includes, for example, facilities fordetermination of one or more, or any of the above disparitydeterminations, and facilities to adjust, control, or turn off any oneor more of various described encoding features (or encoding features tobe developed in the future).

FIG. 6 is a diagram of a video decoder (i.e. AVC decoder) capable ofdecoding optimized coded images/video according to an embodiment of thepresent invention. The decoder is generic, but is fed a signal orotherwise accesses data via a bitstream that is encoded according to thepresent invention. The present invention specifically includes anydecoder, either specialized or generic that is fed or is intended toaccess data (whether transmitted in a bitstream or read from a memorystorage (e.g., memory stick, hard drive, I-pod, etc) encoded accordingto any aspect, feature, device, or process according to the presentinvention.

In describing preferred embodiments of the present invention illustratedin the drawings, specific terminology is employed for the sake ofclarity. However, the present invention is not intended to be limited tothe specific terminology so selected, and it is to be understood thateach specific element includes all technical equivalents which operatein a similar manner. Furthermore, the inventors recognize that newlydeveloped technologies not now known may also be substituted for thedescribed parts and still not depart from the scope of the presentinvention. All other described items, including, but not limited toencoders, decoders, comparitors, multiplexors, processes, processors,arrangements of hardware, including Blu-ray players, patterns, etcshould also be considered in light of any and all available equivalents.

Portions of the present invention may be conveniently implemented usinga conventional general purpose or a specialized digital computer ormicroprocessor programmed according to the teachings of the presentdisclosure, as will be apparent to those skilled in the computer art.

Appropriate software coding can readily be prepared by skilledprogrammers based on the teachings of the present disclosure, as will beapparent to those skilled in the software art. The invention may also beimplemented by the preparation of application specific integratedcircuits or by interconnecting an appropriate network of conventionalcomponent circuits, as will be readily apparent to those skilled in theart based on the present disclosure.

The present invention includes a computer program product which is astorage medium (media) having instructions stored thereon/in which canbe used to control, or cause, a computer to perform any of the processesof the present invention. The storage medium can include, but is notlimited to, any type of disk including floppy disks, mini disks (MD's),optical discs, DVD, HD-DVD, Blue-ray, CD-ROMS, CD or DVD RW+/−,micro-drive, and magneto-optical disks, ROMs, RAMS, EPROMs, EEPROMs,DRAMs, VRAMs, flash memory devices (including flash cards, memorysticks), magnetic or optical cards, SIM cards, MEMS, nanosystems(including molecular memory ICs), RAID devices, remote datastorage/archive/warehousing, or any type of media or device suitable forstoring instructions and/or data.

Stored on any one of the computer readable medium (media), the presentinvention includes software for controlling both the hardware of thegeneral purpose/specialized computer or microprocessor, and for enablingthe computer or microprocessor to interact with a human user or othermechanism utilizing the results of the present invention. Such softwaremay include, but is not limited to, device drivers, operating systems,and user applications. Ultimately, such computer readable media furtherincludes software for performing the present invention, as describedabove.

Included in the programming (software) of the general/specializedcomputer or microprocessor are software modules for implementing theteachings of the present invention, including, but not limited to,preparing a checkerboard of pixels from more than one image, encoding acheckerboard as a frame of a video format, encoding a checkerboard as aframe of a video format and controlling at least one encoding featurefor regions of an image pattern in the checkerboard representing morethan one image where the regions include an amount of disparity in therepresented images that would result in cross-contamination between therepresented images if encoded without controlling the feature, packaginga checkerboard encoded set of images into a transport format, decoding atransport format containing a checkerboard encoded set of images,decoding a video format including a patterned set of images in at leastone frame, decoding a video format including a checkerboard pattern ofdata for images, decoding a video format including a checkerboardpattern of image data representing 1^(st) and 2^(nd) channel image dataof a 3D image, recognizing a patterned set of images in at least oneframe images in at least one frame, and the display, storage, orcommunication of results according to the processes of the presentinvention.

Thus, embodiments of the present invention may relate to one or more ofthe Enumerated Example Embodiments (EEEs) below, each of which areexamples, and, as with any other related discussion provided above,should not be construed as limiting any claim or claims provided yetfurther below as they stand now or as later amended, replaced, or added.Likewise, these examples should not be considered as limiting withrespect to any claim or claims of any related patents and/or patentapplications (including any foreign or international counterpartapplications and/or patents, divisionals, continuations, re-issues,etc.). Examples:

Enumerated Example Embodiment 1 (EEE1). A method of encoding imagescomprising the step of controlling an encoding feature of an encodingprocess for regions of an image pattern representing more than one imageto be encoded by the encoding process when the regions include an amountof disparity in the represented images that would result incross-contamination between the represented images if encoded with thefeature.

EEE2. The method according to EEE1, wherein the step of controlling anencoding feature comprises turning the encoding feature off.

EEE3. The method according to EEE1, wherein the step of controlling anencoding feature comprises using the encoding feature less often thanwhen encoding an image pattern representing a single image.

EEE4. The method according to EEE1, wherein the step of controlling anencoding feature comprises negatively biasing the encoding feature.

EEE5. The method according to EEE1, wherein the represented imagescomprise one of a stereoscopic view, multiple stereoscopic views,multiple views of a same scene, and multiple unrelated views.

EEE6. The method according to EEE1, wherein the step of controllingcomprises a relative degree of change in the encoding feature and therelative degree of change is based on the disparity.

EEE7. The method according to EEE1, wherein the step of controllingcomprises at least partially enabling the encoding feature for regionsdetermined to have zero or near zero disparity and disabling the featurefor all other regions.

EEE8. The method according to EEE1, further comprising the step ofdetermining the disparity by bandpass filtering the images.

EEE9. The method according to EEE8, wherein the bandpass filtering isapplied both vertically and horizontally.

EEE10. The method according to EEE1, further comprising the step ofclassifying the disparity of a region as being one of high disparity,low disparity, and zero disparity, and using the disparityclassification to direct the step of controlling.

EEE11. The method according to EEE1, wherein the amount of disparitycomprises an amount of disparity between neighboring regions in therepresented images.

EEE12. The method according to EEE1, wherein the amount of disparitycomprises an amount of disparity between corresponding regions in therepresented images.

EEE13. The method according to EEE12, wherein the amount of disparitycomprises an amount of stereoscopic disparity.

EEE14. The method according to EEE13, wherein the step of controllingcomprises enabling the encoding feature for a region having relativelylow or no stereoscopic disparity when compared to a corresponding regionof another image in the pattern, and disabling the encoding feature fora region having relatively higher stereoscopic disparity.

EEE15. The method according to EEE1, wherein the disparity comprisedifferences in a location of the regions.

EEE16. The method according to EEE15, wherein the step of controllingcomprises enabling the encoding feature for regions in backgroundlocations and disabling the encoding feature for regions in foregroundlocations.

EEE17. The method according to EEE1, wherein the amount of disparitycomprises an amount of estimated motion in corresponding regions of themore than one image.

EEE18. The method according to EEE17, wherein the estimated motioncomprises a feature based motion estimation.

EEE19. The method according to EEE17, wherein the estimated motioncomprises a pixel based motion estimation.

EEE20. The method according to EEE17, wherein the estimated motioncomprises at least one of a block based, phase based, frequency domain,pixel recursive, true motion region, and Bayesian based motionestimations.

EEE21. The method according to EEE17, wherein the estimated motioncomprises the Enhanced Predictive Zonal Search (EPZS).

EEE22. The method according to EEE17, wherein the step of controllingcomprises at least partially enabling the feature for regions determinedto have zero or near zero motion and disabling the feature for all otherregions.

EEE23. The method according to EEE17, wherein the amount of disparitycomprises an amount of illumination change in the corresponding regions.

EEE24. The method according to EEE23, wherein the corresponding regionscomprise corresponding regions in a stereoscopic pair of images.

EEE25. The method according to EEE1, wherein the amount of disparitycomprises an amount of motion estimation between the images.

EEE26. The method according to EEE25, wherein the motion estimation isdetermined via a search comprising an Enhanced Predictive Zonal Search(EPZS).

EEE27. The method according to EEE1, wherein the encoding featurecomprises one of de-blocking, transform, and quantization.

EEE28. The method according to EEE1, wherein the encoding featurecomprises motion compensation.

EEE29. The method according to EEE1, wherein the encoding featurecomprises at least one of inter-frame and intra-frame prediction.

EEE30. The method according to EEE1, wherein the encoding featurecomprises a transform.

EEE31. The method according to EEE30, wherein the encoding featurefurther comprises quantization.

EEE32. The method according to EEE1, wherein the encoding featurecomprises at least one of a color format and sampling configuration.

EEE33. The method according to EEE1, wherein the encoding processcomprises a video encoding process.

EEE34. The method according to EEE33, wherein the video encoding processcomprises High Definition video encoding.

EEE35. The method according to EEE1, wherein the encoding processcomprises scalable video coding.

EEE36. The method according to EEE35, wherein the scalable video codingcomprises the scalable video coding extension of the AVC/H.264 standard

EEE37. The method according to EEE1, wherein the encoding processcomprises video.

EEE38. The method according to EEE1, wherein the encoding gprocesscomprises any of any of MPEG codecs, including MPEG-1, MPEG-2, MPEG-4Part 2 and MPEG-4 AVC/H.264, or other codecs such as VC1, RealVideo, On2VP6/VP7, or other coding techniques including the Multi-view VideoCoding extension of AVC.

EEE39. The method according to EEE1, wherein the image pattern comprisesa “black” and “white” checkerboard arrangement of pixels where the“black” pixels comprise pixels of an image of a first channel in astereoscopic view and the “white” pixels comprise pixels of a secondchannel in the stereoscopic view.

EEE40. The method according to EEE1, wherein the image pattern comprisesa “multi-colored” “checkerboard” arrangement of pixels where eachindividual “color” of the “checkerboard” comprises pixels of anindividual one of the more than one images.

EEE41. The method according to EEE40, wherein the “multi-colored”“checkerboard” comprises more than two “colors.”

EEE42. The method according to EEE1, wherein the image pattern comprisesa “multi-colored” arrangement of pixels where each individual “color” ofthe arrangement comprises pixels of an individual one of the more thanone images.

EEE43. The method according to EEE42, wherein the arrangement comprisesan arrangement of at least one of rows and columns allocated to each“color”/image.

EEE44. The method according to EEE42, wherein the arrangement comprisesa modulo-spacing arrangement of locations in the arrangement allocatedto each “color”/image.

EEE45. The method according to EEE1, wherein the encoding processcomprises field based encoding.

EEE46. The method according to EEE1, wherein the encoding processcomprises field based encoding performed in at least one of a picturelevel and a macroblock level.

EEE47. The method according to EEE1, wherein the amount of disparity iscomputed via a process comprising at least one of a picture levelanalysis, analysis of a slice, a region level analysis, a macro-block,and block level analysis.

EEE48. The method according to EEE1, wherein the amount of disparity isdetermined in a step comprising computing distortion.

EEE49. The method according to EEE48, wherein the step of computingdistortion comprises separating stereo view samples and computingdistortion based on a 3D view of the separated samples.

EEE50. The method according to EEE48, wherein the computed distortioncomprises at least one of luma and chroma.

EEE51. The method according to EEE48, wherein the computed distortioncomprises a comparison to at least one of the images prior to beingrepresented in the image pattern.

EEE52. The method according to EEE40, wherein the amount of disparity isdetermined in a step comprising computing distortion computed bycomparing the original images to images represented in the“checkerboard” after being decoded.

EEE53. The method according to EEE1, wherein the amount of disparity isdetermined in a step comprising computing distortion wherein thedistortion is computed by comparing original pre-image pattern images tothe images represented in the image pattern after being decoded.

EEE54. The method according to EEE1, wherein the amount of disparity isdetermined in a step comprising computing distortion wherein thedistortion is computed from an original pair of 3D images that aresubsequently encoded into the image pattern and then comparing theoriginal pair of images to the images decoded from the image pattern.

EEE55. The method according to EEE1, further comprising the step ofreducing quality of at least a region of at least one image set of therepresented images.

EEE56. The method according to EEE55, wherein the step of reducingquality is performed based on a content of the image set being reduced.

EEE57. The method according to EEE55, wherein the step of reducingquality is performed based on a pricing model that takes into account adesired and acceptable level of quality of the image sets.

EEE58. The method according to EEE55, wherein the image sets within therepresented images are prioritized based on a desired quality, and thestep of reducing quality, if performed on an image set, is regulatedaccording to the image set priority such that a lower priority image setis reduced in quality further than a higher priority image set.

EEE59. The method according to EEE55, wherein the image sets within therepresented images are prioritized based on a desired quality, and thestep of reducing quality, if performed on an image set, is regulatedaccording to the image set priority and size of the image set such thata lower priority image set is reduced in quality further than a higherpriority image set unless a higher priority image set is too large andadditional space or bandwidth is needed to carry all of the image sets.

EEE60. The method according to EEE55, wherein the step of reducingquality is performed on a view of a stereoscopic image set.

EEE61. A method comprising the steps of:

receiving a stereoscopic pair of images;

combining the images into an image pattern;

evaluating at least one tool for encoding the image pattern; and

applying at least one of the encoding tools in an encoding process if anamount of cross-contamination between the images in either the encodingor corresponding decoding processes is below a predeterminedcross-contamination threshold;

wherein the encoding process comprises an existing video format.

EEE62. The method according to EEE61, wherein the step of applying atleast one of the encoding tools comprises prioritizing regions of theimages and applying the encoding tools to each region according to thepriority.

EEE63. The method according to EEE62, wherein the prioritizationcomprises high priority regions where at least one encoding tool isapplied and low priority regions where the coding tools are notnecessarily applied.

EEE64. The method according to EEE62, wherein the prioritization isbased on a region passing at least one predetermined threshold.

EEE65. The method according to EEE64, wherein the at least onepredetermined threshold comprises at least on of a stereo disparity,motion detection, luminance difference, and chrominance differencethresholds.

EEE66. The method according to EEE61, wherein the existing video formatcomprises a video format not developed for encoding stereoscopic images.

EEE67. The method according to EEE61, wherein the existing video formatcomprises a scalable video coding extension.

EEE68. The method according to EEE61, wherein the step of evaluatingcomprises at least one of stereo disparity analysis, motion estimationanalysis, luminosity analysis, analysis from multi-pass encoding,analysis from pre-processing and earlier passes, and Motion CompensationTemporal Filtering (MCTF) analysis.

EEE69. The method according to EEE61, wherein at least one toolcomprises at least one of a de-blocking tool and a prediction tool.

EEE70. The method according to EEE61, further comprising the step ofapplying at least one of the encoding tools, comprises applying at leastone of the encoding tools at a reduced level if the amount ofcross-contamination is within an acceptable pre-determinedcross-contamination range.

EEE71. The method according to EEE61, wherein the cross-contaminationcomprises at least one of luma, chroma, blocking, and stereo viewcontamination.

EEE72. The method according to EEE61, wherein the step of evaluatingcomprises a cost process.

EEE73. The method according to EEE72, wherein the cost process comprisesa function of computational costs and contamination costs.

EEE74. The method according to EEE72, wherein the cost process comprisesa lagrangian cost comprising

J(λ)=μ_(LV) D _(LV)(R)+μ_(RV) D _(RV)(R)+λ·R

where

D_(LV) and D_(RV) are distortion values for left and right images of theimage pair,

μ_(LV) and μ_(RV) are lagrangian multipliers associated with each image,

R is an estimate or an actual bit-rate for selecting a coding modecomprising the encoding tools, and

λ is a lagrangian parameter for the rate parameter.

EEE75. The method according to EEE74, further comprising the step ofexploiting Stereo View Masking (SVM) by applying at least one of theencoding tools in an encoding process in cases where μ_(LV)>μ_(RV), orμ_(LV)<μ_(RV) resulting in a higher bias towards the quality of oneview.

EEE76. The method according to EEE61, further comprising the step ofexploiting stereo view masking (SVM) by providing a lower qualityversion of one of the stereoscopic pair of images and alternating thelower quality version between subsequent left and right views of thestereoscopic pair of images.

EEE77. The method according to EEE61, further comprising the step ofexploiting stereo view masking (SVM) by providing a varying quality ofat least one of the stereoscopic pair of images wherein the quality isselected based on a priority of the image.

EEE78. The method according to EEE61, further comprising the step ofexploiting Stereo View Masking (SVM) by applying at least one of theencoding tools in an encoding process if an amount ofcross-contamination in one of the images in either the encoding orcorresponding decoding processes is above a first predetermined SVMcross-contamination threshold and an amount of cross-contamination inthe other image in either the encoding or corresponding decodingprocesses is below a second predetermined SVM cross-contaminationthreshold.

EEE79. An encoding device, comprising:

an input port configured to receive a bit pattern comprising image datato be encoded;

an encoder comprising a set of tools configured to encode aspects of theimage data to be encoded;

an evaluation processor configured to, evaluate at least one factorbetween corresponding regions of a set of images embedded in the bitpattern of the image to be decoded, and

regulate an amount of use of at least one of the tools based on saidevaluation.

EEE80. The encoding device according to EEE79, wherein said factorcomprises at least one of disparity, luminance, chrominance, and motionestimation.

EEE81. The encoding device according to EEE79, wherein the set of toolscomprises at least one of a de-blocking tool and a motion predictiontool.

EEE82. The encoding device according to EEE79, wherein the encodercomprises a scalable video encoder.

EEE83. The encoding device according to EEE82, wherein the scalablevideo encoder comprises an extension of an existing video format.

EEE84. The encoding device according to EEE83, wherein the video formatcomprises AVC/H.264.

EEE85. The encoding device according to EEE83, wherein the video formatcomprises one of AVS and VC1.

EEE86. The encoding device according to EEE79, wherein the regulation bythe evaluation processor comprises negatively biasing at least one ofthe tools if at least one of the evaluated factors exceeds apre-determined threshold.

EEE87. The encoding device according to EEE79, wherein the regulation bythe evaluation processor comprises negatively biasing at least one ofthe tools by an amount that varies depending on a priority levelassociated with the evaluated factors.

EEE88. The encoding device according to EEE79, wherein the regulation bythe evaluation processor comprises reducing an effect of use of at leastone of the tools if at least one of the evaluated factors exceeds apre-determined threshold.

EEE89. The encoding device according to EEE79, wherein the set of imagescomprises one of a stereoscopic view, multiple stereoscopic views,multiple views of a same scene, and multiple unrelated views.

EEE90. An add-on device, comprising a decoded video input port, and amulti-view signal output port, wherein the add-on device is configuredto processes a video received on the input port that comprises a decodedmulti-view-in-each-frame video to produce a multi-view signal to betransmitted by the output port.

EEE91. The add-on device according to EEE90, wherein the decodedmulti-view-in-each-frame video comprises a 3D video having left andright views embedded in each frame of the video.

EEE92. The add-on device according to EEE91, wherein at least one of theinput port and the output port comprises a wireless port.

EEE93. The add-on device according to EEE90, wherein the decodedmulti-view-in-each-frame video comprises a video that was decompressedfrom a checkerboard pattern.

EEE94. The add-on device according to EEE90, wherein the decodedmulti-view-in-each-frame video comprises a video that was decompressedfrom a checkerboard pattern and was encoded in the checkerboard patternvia a method that included a step of controlling at least one encodingfeature based on a disparity between at least two of the views of anoriginal video from which that encoding was made.

The present invention may suitably comprise, consist of, or consistessentially of, any of element (the various parts or features of theinvention) and their equivalents as described herein. Further, thepresent invention illustratively disclosed herein may be practiced inthe absence of any element, whether or not specifically disclosedherein. Obviously, numerous modifications and variations of the presentinvention are possible in light of the above teachings. It is thereforeto be understood that within the scope of claims to be included in asubsequently filed utility patent application, the invention may bepracticed otherwise than as specifically described herein.

1. A method of encoding images comprising the step of turning off ornegatively biasing an encoding feature of an encoding process forregions of an image pattern representing more than one image to beencoded by the encoding process when the regions include an amount ofdisparity in the represented images.
 2. The method according to claim 1,wherein the image pattern comprises a checkerboard arrangement of pixelscomprising two channels of a stereoscopic view.
 3. The method accordingto claim 1, wherein the disparity is determined by at least one ofbandpass filtering the image patterns and computing distortion, whereinthe distortion is computed by comparing original pre-image patternimages to the images represented in the image pattern after beingdecoded.
 4. The method according to claim 1, wherein the amount ofdisparity comprises at least one of an amount of estimated motion incorresponding regions of the images, and an amount of illuminationchange in corresponding regions of the images.
 5. An encoding device,comprising: an input port configured to receive a bit pattern comprisingimage data to be encoded; an encoder comprising a set of toolsconfigured to encode aspects of the image data to be encoded; anevaluation processor configured to, evaluate at least one factor betweencorresponding regions of a set of images embedded in the bit pattern ofthe image to be decoded, and regulate an amount of use of at least oneof the tools based on said evaluation.
 6. The encoding device accordingto claim 5, wherein said factor comprises at least one of disparity,luminance, chrominance, and motion estimation.
 7. The encoding deviceaccording to claim 5, wherein the set of tools comprises at least one ofa deblocking tool and a motion prediction tool.
 8. The encoding deviceaccording to claim 5, wherein the encoder comprises a scalable videoencoder.
 9. The encoding device according to claim 5, wherein theregulation by the evaluation processor comprises reducing an effect ofuse of at least one of the tools if at least one of the evaluatedfactors exceeds a pre-determined threshold.
 10. The encoding deviceaccording to claim 5, wherein the images comprises one of a stereoscopicview, multiple stereoscopic views, multiple views of a same scene, andmultiple unrelated views.
 11. A use of a device, comprising a decodedvideo input port, and a multi-view signal output port, wherein thedevice is configured to processes a video received on the input portthat comprises a decoded multi-view-in-each-frame video to produce amulti-view signal to be transmitted by the output port, wherein when thedecoded multi-view-in-each-frame video comprises a 3D video having leftand right views embedded in each frame of the video that had beenencoded by controlling at least one encoding feature based on anevaluation of a factor between the left and right views the deviceproduces a stereoscopic multi-view signal in a format recognized by adisplay connected to the output port, the use being that the device isused to decode the multi-view-in-each-frame video from a checkerboardpattern of pixels comprising the left and the right view.
 12. The use ofthe device according to claim 11, wherein the device is embedded in amedia player.
 13. The use of the device according to claim 11, whereinthe decoded multi-view-in-each-frame video comprises a stereoscopicvideo that was decompressed from the checkerboard pattern whilecontrolling said feature upon evaluation of said factor comprising adisparity between at least two of the views of an original video fromwhich the encoding was made.
 14. An electronic readable media havinginstructions stored thereon that, when loaded into and executed by aprocessing device, cause the processing device to perform the step ofclaim
 1. 15. The method according to claim 1, further comprising thestep of reducing quality of at least a region of at least one image setof the represented images.
 16. The method according to claim 15, whereinthe step of reducing quality is performed based on a content of theimage set being reduced.
 17. The method according to claim 15, whereinthe step of reducing quality is performed based on a pricing model thattakes into account a desired and acceptable level of quality of theimage sets.
 18. The method according to claim 15, wherein image setswithin the represented images are prioritized based on a desiredquality, and the step of reducing quality, if performed on an image set,is regulated according to the image set priority such that a lowerpriority image set is reduced in quality further than a higher priorityimage set.
 19. The method according to claim 15, wherein image setswithin the represented images are prioritized based on a desiredquality, and the step of reducing quality, if performed on an image set,is regulated according to the image set priority and size of the imageset such that a lower priority image set is reduced in quality furtherthan a higher priority image set unless a higher priority image set istoo large and additional space or bandwidth is needed to carry all ofthe image sets.